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Abstract: This research aims to study and assess state-of-the-art physics-informed neural networks 
(PINNs) from different researchers’ perspectives. The PRISMA framework was used for a system- 
atic literature review, and 120 research articles from the computational sciences and engineering 
domain were specifically classified through a well-defined keyword search in Scopus and Web of 
Science databases. Through bibliometric analyses, we have identified journal sources with the most 
publications, authors with high citations, and countries with many publications on PINNs. Some 
newly improved techniques developed to enhance PINN performance and reduce high training 
costs and slowness, among other limitations, have been highlighted. Different approaches have been 
introduced to overcome the limitations of PINNs. In this review, we categorized the newly proposed 
PINN methods into Extended PINNs, Hybrid PINNs, and Minimized Loss techniques. Various 
potential future research directions are outlined based on the limitations of the proposed solutions. 


Keywords: physics-informed neural networks (PINNs); partial differential equation (PDE); loss 
function; activation function; deep learning 


1. Introduction 


Physics-informed neural networks (PINNs) [1] are frequently employed to address a 
variety of scientific computer problems. Due to their superior approximation and general- 
ization capabilities, physics-informed neural networks have gained popularity in solving 
high-dimensional partial differential equations (PDEs) [2]. As a deep learning method, 
physics-informed neural networks bridge the gap between machine learning and scientific 
computing. PINNs have contributed to improvements in many areas of computer science 
and engineering due to their simplicity [3,4]. In the engineering and scientific literature, 
PINNs are receiving more attention for solving a variety of differential equations with 
applications in weather modeling, healthcare, manufacturing, and other fields [5-7]. How- 
ever, PINNs are not suitable for several real-time applications because of their high training 
costs. Although various proposals have been made to enhance the training effectiveness of 
PINNs, only some have considered the effects of initialization [S—10]. Another obstacle to 
the application of PINNs to a wide range of real-world problems is their poor scalability [5]. 

Due to the sheer number of residual points in time-constrained space, an accurate 
network, called a physics-informed neural network (PINN), can be trained by minimizing 
only the residual loss. Moreover, the prediction of the high-fidelity solution for complex 
nonlinear problems in low-dimensional space is more accurate than the solutions of the 
reduced-order equations [1,11—13]. 

The ability of PINNs to learn from sparse input is one of their most well-known 
features [12,14,15]. Initial and boundary terms are not systematically given in the context of 
PINN structure; therefore, they need to be incorporated in the loss function of the network, 
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which must be learned simultaneously with the uncertain functions of the differential 
equation (DE) [16,17]. While implementing gradient-based approaches, combining multiple 
targets during the network’s training can lead to biased gradients, leading to PINNSs failing 
to appropriately learn the fundamental DE solution [18-20]. 

According to Raissi et al. [1], Schiassi et al. [21], and Zhang et al. [22], the key draw- 
back of conventional PINNs is that even the DE limitations are still not mathematically 
solved, hence they need be learned concurrently with the DE solutions within the do- 
main. As a result, we cope with competing goals during PINN training: learning the 
concealed DE solutions well in domains while also learning the hidden DE solutions on 
the boundary [21,23,24]. This results in imbalanced gradients during network training 
using gradient-based approaches, causing PINNs to fail to learn the basic DE solutions 
accurately [21,25]. According to Dwivedi et al. [26], despite the numerous benefits that 
PINNs provide, they have three major drawbacks. The first is their slowness [26,27] when 
applied to real problems; PINNs use up gradient descent optimization and are quite slow 
when compared to other numerical approaches. For highly deep networks, PINNs are 
vulnerable to vanishing gradient problems [6,26,28]. There is also the possibility that a 
solution will become stuck at a minimal point. Finally, the PINN’s learning process is 
fine-tuned by hand. We cannot ascertain exactly how much data or even which framework 
is sufficient for a particular set of sample instances [1,29-31]. 

The weighted least-squares collocation approach utilized in PINNs can be interpreted 
as a hybrid physics/data loss scheme [32-34]. As a result, PINNs have inherited several 
drawbacks common to such approaches, including the necessity to evaluate PDE residuals 
correctly against beginning as well as boundary conditions; a severe regularity demand 
for solutions remains continuous, as does the inability of natural methods to impose 
conservation structures [32,35-37]. 

Although PINNs have been exceptionally beneficial to the scientific community, Colby 
et al. [18] discovered that they are often incapable of appropriately solving a mathematical 
model for interfacial problems used for solidification dynamics called “phase field prob- 
lems”. As a result, they found that specific elements of phase field model solutions (both 
spatially and dynamically) were more difficult to learn than others. These problematic 
places may alter as you discover the solution. 

The goal of this research is to find physics-informed neural network adaptations 
for solving various problems from the literature and to highlight newly improved PINN 
methodologies that have been proposed using different techniques. The main objectives of 
the study were to evaluate the current state of the art in this field of research using numerous 
bibliometric analyses, identify the full spectrum of eligibility requirements studied in the 
literature through information synthesis, create a collection of general information about 
how far research on PINNs has changed over time, and identify newly introduced PINN 
approaches while highlighting some topics for future research. We tried to categorize state- 
of-the-art PINN techniques into three groups. These are Extended, Hybrid, and Minimized 
Loss PINN techniques and will be discussed later in Section 4.1. In this literature review, 
we aimed to answer the following question: What techniques have been introduced to optimize 
the performance of physics-informed neural networks? Although PINNs are utilized to solve 
problems in practically all the domains of human endeavor, throughout this review, we 
focused on computational sciences and engineering. 

The rest of the paper is organized as follows: Section 2 presents the background. 
Section 3 explores the quality assessment and qualitative synthesis used in the literature 
review for this study. Section 4 discusses the results of the bibliometric analysis, as well as 
the objectives, methods, and limitations of the newly proposed PINN techniques. Sections 5 
and 6 discuss future research directions and conclusions, respectively. 


2. Background 


Many methods for solving differential equations have been established over the years. 
Some generate a solution in an array containing the solution’s value in a predefined set 
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of locations. Others employ basis-functions to designate the solution in analytic methods 
and typically translate the genuine problem to a system of algebraic equations. Most past 
projects seeking to solve partial differential equations with neural networks have been 
limited to the case of solving methods of algebraic equations that come from domain 
discretization. In 1998, Isaac Lagaris et al. [38] introduced an improved method for solving 
ordinary DEs and partial differential equations by employing Artificial Neural Networks 
(ANNs). In their newly introduced method, the differential equation’s trial solution was 
represented by the sum of two parts. There were no adjustable parameters in the first half, 
which satisfied the initial/boundary conditions. The second component was designed in a 
way that did not change the initial or boundary conditions. In this section, a feedforward 
neural network (NN) with programmable constraints (the weights) was used. Therefore, 
the network is trained to fit the differential equations together with the initial/ boundary 
conditions that were satisfied by construction [38] methods and can be used for a range 
of ordinary differential equations (ODEs), coupled ODE systems, and partial differential 
equations (PDEs). 

In 2011, Ladislav Zjavka [39] developed a new technique, known as a Differential 
Polynomial Neural Network (D-PNN). The proposed D-PNN technique approximates a 
multi-parametric function by generating and solving unknown partial DEs. A differential 
equation is substituted to create a system model of dependent variables, leading to the 
summation of fractional polynomial derivative terms. In contrast to the ANN method, 
the D-PNN allows each neuron to directly participate in the calculation of the network’s 
overall output. Consequently, in 2013, Ladislav et al. [40] showed that D-PNNs could be 
applied to solve complex mathematical problems. 

In 2015, Ladislav et al. [41] presented a recurrent neural network (RNN) of one layer, 
which is frequently employed for time series predictions and which was used for compari- 
son. With twenty-four succession samples constantly generated by the benchmark and a 
continuous step value of 0.1, in the range of 0—2.4, the D-PNN and RNN were trained. Two 
incredibly different networks were trained with only a relatively narrow range of values, 
which did not accurately reflect function-specific progress over an entire period. The models 
were then tested over a longer period. The networks only estimated one benchmark value 
for the next step x using the calculated accurate function f(x) with three-input sequences, 
rather than making entire predictions built on prior steps’ outputs (approximations). 

In 2017, Raissi et al. [42] proposed hidden physics models (machine learning of non- 
linear partial DEs). To obtain patterns from the high-dimensional data produced by ex- 
periments, the models are essentially data-efficient learning approaches that can exploit 
underlying physical laws expressed by time dependency and nonlinear PDEs. The pro- 
posed methodology can be used to solve learning problems, identify systems, or find partial 
differential equations using data. Within the same time frame, Raissi, Perdikaris, and Karni- 
adakis [43] introduced the prototype of novel physics-informed neural networks (PINNs). 
It was referred to as “data-driven solutions of nonlinear partial differential equations”. The 
term “physics informed neural networks” refers to NNs that have been trained to solve any 
supervised learning tasks while complying by any identifiable physics law as expressed by 
general nonlinear PDEs. They developed two unique kinds of algorithms, namely continu- 
ous time and discrete time models, depending on the type and organization of the available 
data. The result is a new family of universal function approximators that efficiently use 
data and intuitively encode any underlying physical constraints as prior knowledge. 

In 2018, Raissi et al., proposed another technique called “multistep neural networks 
for data-driven discovery of nonlinear dynamical systems” [44]. They presented a machine 
learning strategy for recognizing nonlinear dynamical systems using data. To clarify the 
principles governing the evolution of a particular dataset, they specifically combined 
traditional numerical analysis methods, such as multi-step time-stepping patterns, with 
potent nonlinear function approximators, such as deep neural networks. They described 
how this allowed them to accurately learn dynamics, predict future states, and identify 
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points of attraction. They evaluated their methods for a set of benchmark problems that 
required the identification of complicated, nonlinear, and chaotic dynamics. 

In the same period, Raissi [45] proposed a new technique called Deep-Hidden Physics 
Models to support the use of deep learning in the discovery of nonlinear PDEs. To find 
nonlinear PDEs using dispersed and possibly noisy observations in space and time, they 
proposed a deep learning approach. They used two different deep neural networks to 
estimate the unknown solution and the nonlinear dynamics. They were able to prevent 
numerical differentiations because they were intrinsically unstable and ill-conditioned by 
acting as a prior on the unknown answer. The second network exhibited nonlinear dy- 
namics and aids in our ability to identify the fundamental principles guiding the evolution 
of a certain spatiotemporal dataset. They demonstrated how the developed model could 
help them to grasp the structure and dynamics of a system and forecast its future state by 
testing the efficiency of the approach on a variety of benchmark problems covering a range 
of scientific areas. 

The following are various adjustments to the basic PINN prototype. In early 2019, 
Raissi et al. [1] presented the full version of PINNs as a “Deep Learning Framework for 
Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations”. 

Despite the significant improvement in simulating multi-physics problems by using 
numerical modeling of PDEs, noisy data cannot be totally integrated into current algorithms. 
Mesh creation is still complex and challenging and high-dimensional problems constrained 
by parameterized PDEs cannot be solved [46-48]. Furthermore, inverse problems that 
involve hidden physics are often extremely expensive to solve. This is because they require 
complex computer code and several formulations [49-51]. 

In the recent past, deep learning and physics-informed neural networks (PINNs) have 
garnered a lot of interest in scientific research and engineering applications [52-54]. Deep 
learning has lately evolved as a modern discipline of scientific machine learning in relation 
to solving the governing PDEs of physical situations [55-57] by incorporating the universal 
approximation and high expressivity of neural networks [58-60]. Deep neural networks 
can estimate any high-dimensional function if adequate training data are available [61-63]. 

However, these networks do not consider the physical properties of the problem, and 
the accuracy of the approximation they provide still depends heavily on the exact geometry 
of the problem and the initial and boundary conditions [64-66]. The solution is indeed not 
unique without this basic information, and it may lose physical accuracy. Data are scarce 
and imprecise in most applications, and the controlling physics are unknown, limiting the 
utility of traditional machine learning (ML) and physics-based approaches [56,67—70]. 

Physics-informed neural networks, on the contrary, use fundamental physical equa- 
tions to train neural networks. PINNs should be trained to comply with both the available 
training data and the enforced governing equations. A neural network could be managed 
in this way by training data that need not be huge and full [58,61,71]. 

Large volumes of data are required in a wide range of applications to train the neural 
network through decreasing the distance (loss) in between network output and the ground 
truth [72-75]. 

A PINN’s loss function is made up of numerous terms, such as governing equations 
and boundary conditions. To enable a network model with well-known physical laws, 
Raissi et al. [1] proposed physics-informed neural networks (PINNs), which were intended 
to resolve direct as well as inverse problems controlled by numerous different types of 
PDEs [1,72,76]. 

In partial and ordinary differential equations, PINNs were applied to learn solutions 
and parameters. The basic concept, especially with physics-informed neural networks, is to 
employ physics laws in the form of DEs to train NNs [77]. This is significantly distinctive 
from using the neural networks as dummy models trained with data from a composition 
of input and output values for nonlinear PDEs [1,14,78]. 

The goal is to train neural networks not only on data, as is common in deep learning, 
but also on the fundamental model of the DEs [37,79]. Without understanding the boundary 
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conditions, it is feasible to get a precise solution to PDEs [34,62,80]. Therefore, PINNs can 
be applied to identify an optimum solution of high reliability with some insight on the 
physical properties of the problems and training data (even sparse and partial). 

In other words, PINNs provide an inconsistent method for solving differential equa- 
tions that violates any mathematical discretization of the system [37,81,82]. PINNs show 
improvement in terms of performance and accuracy while only using a small amount of 
data for training. They are used to perfectly describe the physical attributes of a system’s 
dynamic ecosystem [83-85]. 

PINNs provide the solutions to a broad range of computational science problems and 
are a pioneering technology that is leading towards the advancement of new categories 
of numerical solvers for PDEs. PINNs can be viewed as a meshless alternative to classic 
methodologies (for example, CFD for fluid dynamics) and novel data-driven methods for 
model inversion and system identification [1,3,86]. 

Moreover, the trained PINNs could predict values on modeled grids of various reso- 
lutions without having to be retrained [87]. In addition, they allow for the manipulation 
of automatic differentiation (AD) [88,89] when calculating the appropriate derivatives in 
PDEs, a novel class of differentiation techniques frequently utilized to derive NNs that have 
been shown to be exceptional regarding numeric or symbolic differentiation [87,88,90]. 


2.1. Physics-Informed Neural Networks 


PDEs with high dimensions are commonly used in a variety of disciplines, including 
physics, chemistry, engineering, finance, and more [91]. Higher dimensions make numerical 
PDE computational methods such as finite difference or finite element methods impractical 
due to the explosion in the number of grid points and the need for smaller time steps. 

Physics-informed neural networks (PINNs) are models developed to obey physical 
laws specified by (nonlinear) partial differential equations (PDEs). They can be used for 
supervised tasks where the reduction of errors with respect to data and physical laws is 
required [1,91,92]. 

The loss function is defined as follows: 


1 = I gata + W physics (1) 


where I gaiq is the loss with respect to the data, wl physics 18 a physics law’s loss relative to the 
PDEs, and w is the weight matrix of a physics loss. By using mean-squared error for both 
losses, if f; is the error of each physical state, we derive: 


l=). i-9° -whf? (2) 


The input values x, y, and t are used in the architecture of the network, and the output 
values are u and v. The output variables are used directly to calculate the data loss term. 
However, we distinguish the variables with respect to the input variables for the physical 
loss term to account for the physical loss function, as shown in Figure 1 [91] below. The 
goal is to minimize deviations from the physics law by monitoring training with measured 
or generated data (i.e., Iagtq), as well as by training to diminish the departure from the 
physics law. To further train to minimize deviation from the physical law, by preventing 
overfitting, the term / physics eNSUres that such a neural network generalizes better given 
unknown inputs. The physical loss allows the output variables to be trained to include 
their first and/or second-degree derivatives, as well as a local region around each input 
value of the given data (depending on the PDEs). 
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Figure 1. Schematic diagram of a PINN. Adapted with permission from Ref. [91]. Copyright 2021 De 
Wolff, Carrillo, Marti, and Sanchez-Pi. 


2.2. Modeling and Computation 
A general nonlinear partial differential equation can be: 


ou 
ap 
ur +N{u;A] = 0, x €O, te [0,T] (3) 


u(t,x) denotes the solution, N[u; A] is a nonlinear operator parametrized by A, and Nisa 
subset of R?. 

Numerous problems in mathematical physics, such as conservative laws, diffusion 
processes, advection—diffusion systems, and kinetic equations, fall under this general 
category of governing equations. 

PINNs can be programmed to solve two major types of PDE problems on the noisy 
data of a general dynamical system denoted by the above equation [14,92,93]. These are: 


1. Data-driven solutions. 
2. Data-driven discovery. 


e Data-Driven solutions of Partial Differential Equations 
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The data-driven solution of PDE findings when calculating the unknown state u(t, x) 
of the system given noisy measurements z of the state and fixed model parameters A reads 
as follows: 

ur +N[u] =0, xeQ, te [0,T] (4) 


By defining f (t,x) as 
f:=um+N[u] =0 (5) 
and approximating u(t,x) with a deep neural network, f(t,x) results in a PINN. This 


network can be differentiated using automatic differentiation. The parameters of u(t, x) 
and f (t,x) can be then learned by minimizing the following loss function Lio: 


Ltot = Lu + Lr (6) 
where L, = | | u—zl | 7 with u and z representing state solutions and measurements at 
sparse location T, respectively, and Ly = | |f | | 7 is a residual function. To satisfy this second 


term, the structured data defined by the PDEs must be used during the training phase [81]. 

This approach enables the creation of physically informed surrogate models that are 
computationally efficient and which can be used for simulations, model predictive control, 
and data-driven physical process predictions [1,94—96]. 


e Data Discovery of Partial Differential Equations 


The data-driven discovery of PDEs leads to computation of the unknown states u(t, x) 
and f (t,x) of the system and the learning of model parameters, A, that most accurately 
capture the observed data: 


uz + Niu; A] =0, xe O, te [0,T] (7) 
Additionally, f(t, x) is defined as 
f:=ur+N[u;A] =0 (8) 


Further, approximating u(t,x) with a deep neural network, (t,x), leads to a PINN. 
This network can be derived by automatic differentiation. The parameters of u(t,x) and 
f (t,x) can then be learned along with the parameters, A, of the differential operator by 
minimizing the following loss function Lio:: 


Ltot = Ly + Lr (9) 
where L,, = | |u—zl 17 with u and z representing state solutions and measurements at 
sparse location T, respectively, and Ly = | If | | 7 is a residual function. This second term 


involves the well-defined information characterized by the PDEs to be satisfied in the 
training procedure. 


3. Methodology 


For reviewing current research, this paper used the PRISMA framework [97-99]. The 
scoping approach was utilized to retrieve the most relevant papers on physics-informed 
neural networks. This method aided the control of critical mandatory components and the 
classification of potential search terms [97,100,101]. To identify relevant scientific papers 
and articles, multiple databases were searched. A search using a single keyword (“physics- 
informed neural networks”) was conducted to find relevant publications from the most 
reputable and reliable research resources. Scopus and Web of Science were used, along with 
the Web of Science core collection, Derwent Innovations Index, MEDLINE, KCI-Korean 
Journal Database, and SCIELO Citation Index. 

The keyword “Physics Informed Neural Network” was solely used in each database 
search for the relevant literature. Predefined inclusion and exclusion criteria and quality 
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requirements were used to refine the data search. Each filter verified that the quality 
requirement was met, and the next section discusses the inclusion and exclusion criteria. 

Because our search query was put in a double quote, we employed deterministic 
information retrieval to look for suitable papers, as we described earlier. The literature 
searches in all the databases listed above retrieved articles from 2019 to 2022. Initially, 530 
items were found; however, this was largely made up of a variety of materials, such as 
research articles, reviews, editorials, and book chapters, among others. 

Many researchers use PINNs to solve problems in different areas of human endeavor. 
We have limited our research to computational sciences and engineering and focused on 
research articles, review papers, and book chapters in our literature search. A total of 288 
documents were chosen, as illustrated in Figure 2. Articles from computational sciences 
and engineering were chosen as the following sequence. This included computer science, 
engineering, mathematics, and physics. Non-English documents were also excluded. The 
PRISMA checklist can be found in Supplementary Materials. 


Addition of 
records from 
databases 


405+125 = 530 «_— Physics Informed 


Neural Network” 


Filtration in 203+85 = 288 


databases 
92+69 = 161 Removal of duplicate 
/ articles and eligibility 
assessment 
Articles included for 
quantitative synthesis 
Articles included for 
bibliometric analysis 


Figure 2. PRISMA framework. 


Duplicate articles were also removed from the Scopus and Web of Science metadata 
files. The two files were combined and duplicate journal articles removed. In this review, 
the PRISMA 2020 flow diagram was used, as shown in Figure 3 below. 
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Records removed before 
screening: 
Duplicate records removed 
(n=30) 
Records marked as ineligible 
by automation tools (n =20) 
Records removed for other 
reasons (n =5) 


Records identified from*: 
Databases (n =2) 
Registers (n =2) 


Records excluded before 
Records screened retrieval. Deep Learning, 
(n =288) Machine Leaming, PDE, and 
Physics journals 
(n = 166) 


Reports sought for retrieval ; 
(n =161) Reports not retrieved 


(n = 26) 


Reports assessed for eligibility 

(n =135) Reports excluded: 
Journals that are not related 
to Computational Science 
and Engineering (n=15) 


Studies included in review 
(n =120) 

Reports of included studies 
(n =120) 


Figure 3. PRISMA flow diagram. 


3.1. Quality Assessment 


This review looks at final published journal articles, reviews, and conference papers 
to find the best results and capture an excellent overview of previous data. Abstracts and 
conclusions were separated to keep the archive to a minimum. In addition, cited references 
in the evaluated articles were considered. As stated earlier, the two metafile records were 
combined. The duplicate records were eliminated to improve the findings. Irrelevant data 
were also excluded. 


3.2. Qualitative Synthesis Used in the Literature Review 


After selecting the documents, a two-step approach was used to confirm the quality 
of the analysis performed on the published papers. The relevant metadata were initially 
imported into Microsoft Excel to conduct a descriptive study of physics-informed neural 
network literature, which included identification of articles relating to the evolution and 
improvement of PINNs in computational sciences and engineering, among others. 

Content analysis was performed in the second stage to classify and analyze recent 
research across different disciplines and highlight potential challenges and limitations 
which could represent opportunities for future research. 


3.3. Quantitative Synthesis (Meta-Analysis) 


In a systematic review, quantitative synthesis is used to present statistical data. Typi- 
cally, this is referred to as a meta-analysis. Table 1 below is the quantitative study character- 
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istics table, which consists of three groups. Journals are grouped by specialization, type, 
and the PINN method used in those papers. 


Table 1. Quantitative study characteristics. 


Group Population Percentage (%) 


Journal by Specialization 


1. Computer Science 29 24.167 
2. Engineering 31 25.833 
3. Mathematic 35 29.166 
4. Physics 25 20.833 
Journal by Type 
1. Conference Article 21 17.500 
2. Journal Article 99 82.500 
Journal by Methods 
1. Conventional PINNs 97 80.833 
2. Extended PINNs 12 10.000 
3. Hybrid PINNs 7 5.833 
4. Minimized Loss PINNs 4 3.333 


4. Result of Bibliometric Analyses 


The main objectives of this research, as mentioned earlier, were to assess the state of 
the art in this area of study using various bibliometric analyses and to classify the full range 
of eligibility requirements analyzed in the literature through metadata synthesis to produce 
a collection of general information on the extent to which the study of PINNs has changed 
over time. 

Figure 4 illustrates physics-informed neural network evolution over the past three and 
a half years in terms of the number of publication relating to this area of study. Between 
2019 and mid-2022, research in this field had been developing steadily, with a current peak 
in 2021. The trend shows that many researchers are busy finding solutions with PINNs, 
and some are also trying to optimize performance. 


Publications Per Year 


a [I 


2019 2020 2021 2022 


Figure 4. Publications per year. 


Although PINNSs represent a new field of research, they have attracted the interest 
of many researchers around the world. Figure 5 shows the countries with the most PINN 
publications. The United States, China, and European countries have the most publications. 
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Countries With Most Publications 


Others 
23% 


USA 
29% 


Singapore 
3% 
Italy 
2% 
United 
Kingdom 
4% 
Switzerland 
4% 


Germany 
10% 25% 


Figure 5. Countries with the most publications. 


The seven most common publication sources represented in the final paper set are 
summarized in Figure 6. The data show that Computer Methods in Applied Mechanics 
and Engineering is the journal with the most articles in this area of study, followed by 
the Journal of Computational Physics. Many journals and conference proceedings have 
published numerous papers on PINNs. Out of 120 papers selected for this study, we 
considered journals with at least 3 published works in the list of article sources with the 
most publications. 

The citation report of the studies from 2019 to mid-2022 is presented in Table 1. The 
most-cited authors are Raissi M., Perdikaris P., and Karniadakis G.E. [1], with 3442 citations 
as at the time of writing this paper. The most cited article’s title is “Physics-informed 
neural networks: A deep learning framework for solving forward and inverse problems 
involving nonlinear partial differential equations”. It was published by the Journal of 
Computational Physics and is indexed in the Scopus and Web of Science databases. These 
authors are considered pioneers of PINN research. After that, the second most-cited authors 
are Yang, Liu et al. [102] and Goswami et al. [103]. Their works were cited 183 and 177 times 
and published by the Journal of Computational Physics and Theoretical and Applied Fracture 
Mechanics, respectively. Further commonly cited studies are shown in Table 2 below. 

Figure 7 illustrates journal publication sources with the most citations relating to 
physics-informed neural networks. No other journal publisher matched the number of 
citations that the Journal of Computational Physics received. The second most cited journals 
were Communication in Computational Physics and Computer Methods in Applied Mechanics 
and Engineering. 
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Figure 6. Journals with the most publications. 
Table 2. Authors with the most citations. 
Authors Source Title Number of Citations 
Raissi et al. [1] Journal of Computational Physics 3442 
Costabal et al. [104] Frontiers in Physics 122 
Communications in Computational 
Jagtap A.D. et al. [105] Physics 118 
: Computer Methods in Applied 
Meng, subunetal 0G] Mechanics and Engineering oa 
Yang, Liu et al. [102] Journal of Computational Physics 183 
‘ Computer Methods in Applied 
Hoghighat®: etal 110% Mechanics and Engineering nee 
Kharazmi E. et al. [108] ee eer 1 
Mechanics and Engineering 
Fang, Yin et al. [109] Nonlinear Dynamics 29 
Doutado:A, etal, [0] Journal of Computing and Information 37 
Science in Engineering 
Shin Y. et al. [111] Communications in Computational 137 


Physics 
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Table 2. Cont. 


Authors 


Source Title 


Number of Citations 


Engineering Applications of Artificial 


Zobeiry N. et al. [112] Intelligence 52 
Goswami'ét al: [103] Theoretical And Applied Fracture 177 
Mechanics 
Mehta, Pavanetal [118] Fractional Calculus and Applied 25 
Analysis 
Colby et al. [18] Communications in Computational 54 
Physics 
: nee Computer Methods in Applied 
Han, Minhene etal AeA Mechanics and Engineering 2a 
Doan N.A.K. et al. [82] Journal of Computational Science 28 
Rao, Chengping et al. [92] Journal of Engineering Mechanics 57 
Pu, Juncai et al. [115] Nonlinear Dynamics 21 
Meng, Xuhui et al. [116] Journal of Computational Physics 31 
: Computer Methods in Applied 
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PINN research has garnered support from researchers and technology practitioners 
in recent years, as shown in Figure 4. Despite the popularity of PINNs, contributions to 
the field come from few countries around the globe. A review of the number of countries 
with the most publications related to PINNs using bibliometric analysis could spark the 
interest of scientists and technology practitioners from other countries and encourage 
them to collaborate and contribute, as many developed and developing countries are 
now competing for higher rankings in the lists of the best countries for scientific and 
technical research. 

Consequently, evaluating journals with the most publications and authors with the 
most citations through bibliometric analysis can lead to more contributors to the literature 
because many journal publishers may emphasize PINN research. The number of citations 
an author receives from a particular article portrays how other people acknowledge their 
contribution and the impact of their research in the academic domain. 

In this study, we found that highly cited journals have a unique novelty in terms of 
the area of discussion. They try to solve different scientific and technical problems with the 
techniques they propose. The publication from Raissi et al. [1] is the most influential article 
among all others. As we mentioned earlier, they are the pioneers of PINNs and the results 
of their work laid the foundation for PINN research. 

In short, bibliometric analysis with a systematic literature search would give enthu- 
siasts an insight into how far PINN research has progressed and where they should start 
their research. 


4.1. Newly Proposed PINN Methods 


One of the main contributions of our study was finding a solution to the limitations of 
PINNs as mentioned by some authors. Our study also highlights several issues for future 
research. To focus on the research question, we evaluated the work of numerous authors 
who have worked on improving the performance of PINNs and found solutions to many 
of the limitations previously mentioned by different authors. 

Although PINN architecture is built based on feedforward neural networks, due to 
the shortcomings of PINNs many authors have tried to extend PINNs by using different 
approaches, such as conservative PINNs (cPINNs), nonlocal PINNs (nPINNs), etc. In 
contrast, others have tried to combine it with other neural network techniques, expecting 
better performance and more precise results. This includes CNNs, RNNs, etc. Consequently, 
other authors have tried to boost performance by reducing loss to a minimal level. We 
tried to group newly proposed techniques into three categories: Extended PINNs, Hybrid 
PINNs, and Minimized Loss techniques. 


4.1.1. Extended PINNs 


The studies under this category tried to extend the PINN using various techniques, 
such as domain and subdomain decomposition, to improve the generalization of PINNs. 
Other techniques, such as PINNs with ensemble methods, tried to expand the solution 
interval of PINNs to converge to the correct solution. Bayesian physics-informed neural 
networks (B-PINNs) are a type of Extended PINN that is more accurate and much faster 
than a simple PINN. The objectives and limitations of many more techniques are briefly 
discussed in Table 3 below. 
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Author 


Table 3. Proposed Extended PINNs. 


Objective(s) 


Technique 


Limitation(s) 


Jagtap A.D. et al. [35] 


Jagtap A.D. et al. [117] 


De Ryck et al. [14] 


G. Pang et al. [118] 


Liu Yang et al. [102] 


Ehsan Kharazmi et al. [108] 


Juncai Pu et al. [115] 


Enrico Schiassi et al. [21] 


Rafiq et al. [119] 


Gaétan et al. [120] 


Colby et al. [18] 


Katsiaryna et al. [121] 


The main goal of this study was to 
develop a unique conservative 
physics-informed neural network 
(cPINN) for solving 
complicated problems. 


The main objective of this study was 
to introduce an XPINN model that 
improved the generalization 
capabilities of PINNs. 


The main goal of this study was to 
precisely constrain the errors arising 
from the use of XPINNs to 
approximate incompressible 
Navier-Stokes equations. 

This study aimed to extend PINNs to 
the inference of parameters and 
functions for integral equations, such 
as nonlocal Poisson and nonlocal 
turbulence models (nPINNs). A wide 
range of datasets must be adaptable to 
fit the nPINNs. 

The aim of this study was to introduce 
a novel method that was designed for 
solving both forward and inverse 
nonlinear problems outlined by PDEs 
with noisy data, which aimed to be 
more accurate and much faster than a 
simple PINN. 

The purpose of this research was to 
bring together current developments 
in deep learning techniques for PDEs 
based on residuals of least-squares 
equations using a newly 
developed method. 

The goal of the study was to provide 
an improved PINN approach for 
localized wave solutions of the 
derivative nonlinear Schrédinger 
equation in complex space with faster 
convergence and optimum 
simulation performance. 

The main objective of this study was 
to propose a novel model for 
providing solutions to problems with 
parametric differential equations (DEs) 
that is more accurate and robust. 
The main goal of this experiment was 
to propose a unique deep Fourier 
neural network that expands 
information using spectral feature 
combination and a Fourier neural 
operator as the principal component. 
The major objective of this experiment 
was to design a robust model 
architecture for reconstructing 
periodic flows with a small number of 
imperfect sensors by extending PINNs 
with forced truncated 
Fourier decomposition. 

The primary objective of this study 
was to present an Extended PINN 
method which is more effective and 
accurate in solving larger 
PDE problems. 


The objective of this experiment was 
to determine an acceptable time 
window for expanding the solution 
interval using an ensemble of PINNs. 


Conservative physics-informed 
neural network (cPINN) 


Extended physics-informed 
neural networks (XPINNs) 


PINN error estimates 


Nonlocal physics-informed neural 
networks (nPINNs) 


Bayesian physics-informed neural 
networks (B-PINNs) 


Variational physics-informed 
neural networks (hp-VPINNs) 


Improved PINN method 


Physics-informed neural network 
theory of functional connections 
(PINN-TFC) 


Deep spectral feature aggregation 
physics-informed neural network 
(DSFA-PINN) 


Modal physics-informed neural 
networks (ModalPINNs) 


Adaptive physics informed 
neural networks 


PINNs with ensemble models 


Despite the parallelization of the 
cPINN, it cannot be used for 
parallel computation. 


XPINNs enhance generalization in 
exceptional conditions. 
Decomposition results in less 
training data, which makes the 
model more likely to overfit and 
lose generalizability. 


The authors’ estimates in their 
experiment gave no indication of 
training errors. 


nPINNs require more residual 
points. Increasing the number of 
discretization points, on the other 
hand, makes optimization more 
challenging and ineffective, and 
causes error stagnation. 


The proposed B-PINNs in this 
work were only tested in scenarios 
where data size was up to several 
hundreds, and no tests were 
performed with large datasets. 


Although VPINN performance on 
inverse problems is encouraging, 
no comparison was made to 
classical approaches. 


Complex integrable equations 
were not really considered in 
this study. 


The proposed technique cannot be 
applied to data-driven discovery 
of problems when solving ODEs 

using both a deterministic and 

probabilistic approach. 

Other mathematical functions, 

such as the Laplace transform 
coupled with a Fourier transform, 
as well as the conventional CNN, 

cannot be used to generalize 
models using this method. 


The application of ModalPINNs is 
restricted to fluid mechanics only. 


This study focused primarily on 
the problem of solving 
differential equations. 


The ensemble algorithm seems to 
be more computationally 
intensive than the standard PINN 
and is not applicable to 
complex systems. 
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4.1.2. Hybrid PINNs 


Although PINNs are feedforward in nature, some researchers have tried to combine 
them with a variety of neural network architectures to overcome their limitations and 
improve overall performance. Parareal physics-informed neural networks (PPINNs) were 
proposed to improve high-level computational efficacy by using a small dataset. Hybrid 
physics-informed neural network (Hybrid PINNs) use convolutional neural networks to 
solve PDE problems. Physics-informed recurrent neural networks model an industrial 
process called a grey box. The objectives and limitations of Hybrid PINNs are highlighted 
in Table 4 below. 


Table 4. Proposed Hybrid PINNs. 


Author 


Objective(s) 


Technique 


Limitation(s) 


Meng et al. [106] 


Zhiwei Fang et al. [33] 


Lahariya M. [122] 


Wengian Chen et al. [11] 


Xiaoping Zhang [123] 


Dourado et al. [110] 


Mingyuan Yang [124] 


The main goal of this research was to 
introduce a new a hybrid technique 
that can exploit the high-level 
computational efficacy of training a 
neural network with small datasets to 
significantly speed up the time taken 
to find solutions to challenging 
physical problems. 

This paper aimed to present a Hybrid 
PINN for PDEs and a differential 
operator approximation for solving 
the PDEs using a convolutional neural 
network (CNN). 

The goal of this research was to 
propose a physics-informed neural 
network based on grey-box modeling 
methods for identifying energy 
buffers using a recurrent 
neural network. 

The main goal of this research was to 
develop a reduced-order model that 
uses high-accuracy snapshots to 
generate reduced basis information 
from the accurate network while 
reducing the weighted sum of 
residual losses from the 
reduced-order equation. 

The main objective of this study was 
to develop a novel method for solving 
groundwater flow equations using 
deep learning techniques. 

The major goal of this experiment was 
to develop a hybrid technique for 
missing physics estimates in 
cumulative damage models by 
combining data-driven and 
physics-informed layers in deep 
neural networks. 

The goal of this experiment was to 
develop a new hybrid model for 


uncertain forward and inverse 
PDE problems. 


Parareal physics-informed neural 
network (PPINN) 


Hybrid physics-informed neural 
network (Hybrid PINN) 


Physics-informed recurrent 
neural networks 


Physics-reinforced neural 
network (PRNN) 


Ground Water-PINN (GW-PINN) 


PINNs for missing physics 


Multi-Output physics-informed 
neural network (MO-PINN) 


4.1.3. Minimized Loss Techniques 


Domain decomposition of 
fundamental problems with huge 
spatial databases cannot be solved 

with PPINNs. 


This Hybrid PINN is not 
applicable to nonlinear operators. 


The proposed model was not 
validated with real-world 
industrial processes. 


The reduced basis set must be 
small to outperform the 
Proper Orthogonal 
Decomposition—Galerkin 
(POD-G) method in terms of 
accuracy, as the numerical results 
of the experiment showed. 


The proposed model cannot be 
used to predict groundwater flow 
in more complex and larger areas. 


Even if the proposed additional 
levels are used to initialize the 
neural network, suboptimal 
setting of these parameters may 
lead to the failure of the training. 


The proposed method cannot be 
used to solve problems involving 
multi-fidelity data. 


The newly proposed methods under this category attempt to minimize loss using 
different techniques. For instance, the new reptile initialization-based physics-informed 
neural network (NRPINN) uses many sample tasks from parameterized PDEs. It modifies 
the loss penalty term so the model can adopt the initialization parameters of related 
tasks using supervised, unsupervised, and semi-supervised learning. Consequently, the 
physics-informed and physics-penalized neural network model (PI-PP-NN) expresses 
many physical constraints and integrates the regulating physical laws into its loss function. 
The objectives and limitations of newly proposed techniques are briefly discussed in 
Table 5 below. 
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Table 5. Proposed PINN techniques using Minimized Loss. 


Author Objective(s) Technique Limitation(s) 


The main goal of this study 
was to provide a 


gradient-based meta-learning Optimizing the performance 
method for offline discovery of methods such as RMSProp 
that uses data from task Meta-learning PINN and Adam for handling inner 
Apostolos F. et al. [125] agen : c oe ‘ 
distributions created using loss functions optimizers with memory was 
parameterized PDEs with not considered in 
numerous benchmarks to this experiment. 


meta-learn PINN 
loss functions. 
The major objective of this 
experiment was to use 
multiple sample tasks from da aeislicationcbnedd NRPINNs cannot be used to 
Liu X. et al. [8] parameterized PDEs and physics-informed neural solve problems in the absence 
modify the loss penalty term . twork (NRPINN) of prior knowledge. 
to introduce a novel method 
that depends on labeled data. 
The main goal of this 
experiment was to develop a 
model that expresses physical 
constraints and integrates the 


New reptile 


The proposed model can only 
be used to create friction 


sae wiweical Lawes asia Physics-informed and pendulum bearings. For any 
Habib et al. [126] si oe fe Gon physics-penalized neural other isolation system, the 
ee ee network model (PI-PP-NN) theoretical basis must be 


(physics-informed), which the 
model penalizes when they 
are violated 
(physics-penalized). 


adapted accordingly before it 
can be used for design. 


In this experiment, the 
The main goal of this adaptive weight of PDE loss 
experiment was to develop a gradually decreased. 


technique that allows PINNs sab aus Therefore, a theoretical 


Zixue Mang [27] to perfectly and efficiently physics-informed netral investigation of this paradigm 
: : networks (IbPINNs) . . 
learn PDEs using Gaussian is necessary to increase the 
probabilistic models. robustness and scalability of 
the technique. 


5. Future Research Direction 


We previously pointed out various limitations of PINNs that were highlighted by 
different authors and later discussed the newly proposed techniques used to solve most 
of the mentioned problems. We have also outlined the limitations of the newly proposed 
PINN methods. The focus of our future research will be more on the limitations of the 
newly proposed techniques. Since they have addressed most of the PINN limitations, the 
limitations of some selected articles are discussed because of their significance. One of the 
main benefits of using the cPINNs introduced by [35] to solve complicated problems is 
their capacity for parallelization, which effectively lowers training costs; however, they 
cannot be used for parallel computation. In future research, cPINNs may be extended for 
use in parallel computation. 

The enhanced PINN technique proposed by [115] improves neural network perfor- 
mance. Through numerous trainable parameters in the activation function, they noted 
that an important issue for the future was considering how to integrate machine learn- 
ing using integrable systems theory more fully and construct substantial integrable deep 
learning algorithms. 

A new approach presented by [21] can also be applied to data-driven discoveries and 
solutions of parametric differential equations. Future research will focus on extending this 
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method to data-driven problem discovery for solving ODEs using both deterministic and 
probabilistic methods. 

Noting the nature of optimization when training PINNs, the capacity to perform 
uncertainty quantification (UQ) on physical systems was highlighted by [1]. The PINN 
framework implemented by [92] could be extended to provide these features. 

Despite the numerous desirable features of the hybrid approach proposed by [110], 
their findings proved that the methodology has a shortcoming that could be considered as 
a topic of future research. Their research has limitations regarding training results, as they 
were proven to be very responsive to the initial hyperparameters of the neural network; 
even if the proposed auxiliary planes are used to initialize the neural network, the poor 
initialization of these parameters can potentially hinder training. Therefore, optimization 
with a wide range of initial hyperparameter values is still required. 

The recent work of Liu X. et al. [8] has one limitation. As a new reptilian initializa- 
tion learning task, NRPINN needs prior information, especially higher- and zero-order 
information. As a result, NRPINN is not suitable for handling problems where previ- 
ous information is not available. Using transfer learning from related efforts to acquire 
an initialization may be another technique for improving the performance of PINNs in 
future studies. 

A new hybrid technique proposed by [106] cannot be used to solve the domain 
decomposition of basic problems with big spatial databases. Extending this work to support 
large spatial databases could be a possible area of future research. The Ground Water-PINN 
(GW-PINN) technique proposed by [123] can be extended to predict groundwater flow in 
more complex and larger areas. 

The work of [124] could be extended to problems with multi-fidelity data in future 
research. Additionally, their approach could also be stretched to time-dependent problems 
because time dimension is comparable to space dimensions from an implementation 
standpoint; since early uncertainties are transmitted and magnified over time, distinct 
behavior in relation to the uncertainty correlation may emerge. 


6. Conclusions 


The review and bibliometric analysis of the published literature revealed several 
limitations of PINNs. We discovered that a significant amount of experimental research 
in the 120 peer-reviewed articles used conventional PINNs to solve various scientific and 
engineering problems. In contrast, other studies developed new methods to overcome the 
limitations of PINNs and achieve higher performance results. 

Data evaluation was an essential and significant step in the review. Based on the 
relative number of PINN-related journals, two reputable databases were chosen: Scopus 
and Web of Science. The first PINN papers were published in early 2019. For this reason, 
the papers used in this study are from 2019 to mid-2022. 

We discussed the objectives, methodologies, and limitations of the newly proposed 
PINN techniques. Despite the feedforward nature of PINNs, we have found several arti- 
cles that combined them with other neural network architectures. While some extended 
the conventional PINNs, others tried to improve performance by using different tech- 
niques for reducing loss. We have also identified several potential research directions for 
PINNs based on the limitations of proposed solutions to increasing prediction power and 
optimizing performance. 

As part of our contribution to the literature, we intend to implement a new model that 
combines PINNs with either a graph neural network or a recurrent neural network using a 
time series dataset. 


Supplementary Materials: The following supporting information can be downloaded at: https: 
//www.mdpi.com/article/10.3390/bdcc6040140/s1, the PRISMA 2020 Checklist [128]. 
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